Real-world calibration and transportability of the Disease Recovery Evaluation and Modification (DREaM) randomized clinical trial in adult Medicaid beneficiaries with recent-onset schizophrenia

BACKGROUND: The Disease Recovery Evaluation and Modification study (DREaM; NCT02431702) assessed the benefit of initiating paliperidone palmitate (PP), a long-acting injectable antipsychotic, in patients with recent-onset schizophrenia or schizophreniform disorder. OBJECTIVE: To determine whether reductions in psychiatric hospitalizations with early initiation of PP vs oral antipsychotic (OAP) therapy observed in a DREaM post hoc analysis are transportable to a real-world population of patients with recent-onset schizophrenia. METHODS: Patients enrolled in DREaM were randomized to receive OAP or PP for 9 months, after which OAP recipients were re-randomized to receive OAP or PP for another 9 months. We used this design to form treatment arms: OAP-OAP, OAP-PP, and PP-PP. Inclusion/exclusion criteria were used to identify a Medicaid Managed Care (MMC) OAP-treated cohort of 1,000 patients diagnosed with schizophrenia using IBM Truven databases from 2015 to 2019. The MMC cohort was combined with the subset of patients diagnosed with schizophrenia enrolled in DREaM from US sites (N = 45, 43, and 44 for OAP-OAP, OAP-PP, and PP-PP, respectively). Propensity scores for the MMC cohort were estimated using baseline variables identified via double-lasso regression. Estimated propensity scores were used to weight psychiatric hospitalizations in the DREaM OAP-OAP group and compared with observed MMC OAP cohort psychiatric hospitalizations. After the successful calibration of the DREaM OAP-OAP group, similar approaches were taken for the OAP-PP and PP-PP groups to transport DREaM effects to MMC data. RESULTS: Standardized mean differences in baseline covariates between DREaM treatment arms and MMC groups were substantially reduced after calibration. The 18-month cumulative numbers of psychiatric hospitalizations per patient (SE) were 0.83 (0.14) for the MMC cohort, 0.43 (0.14) for the unweighted OAP-OAP, and 0.80 (0.37) for the calibrated OAP-OAP. The difference between the calibrated OAP-OAP and MMC was not statistically significant (difference, 0.03 [95% CI = −0.67 to 0.81]), indicating successful calibration. The mean difference in 18-month cumulative psychiatric hospitalizations relative to the MMC cohort was -0.77 (95% CI = −1.08 to −0.47) for OAP-PP and −0.83 (95% CI = −1.15 to −0.60) for PP-PP. CONCLUSIONS: Our study demonstrates that results from the DREaM OAP-OAP group reflect psychiatric hospitalizations in a real-world population when calibrated using specific baseline characteristics. Transporting the DREaM effects, we find that using OAP-PP and PP-PP treatment strategies for patients with recent-onset schizophrenia in the MMC population could reduce psychiatric hospitalizations compared with the use of OAP. These findings, along with the potential reduction in associated costs, should be considered when assessing the value of PP formulations.

diagnosed with schizophrenia enrolled in DREaM from US sites (N = 45, 43, and 44 for OAP-OAP, OAP-PP, and PP-PP, respectively). Propensity scores for the MMC cohort were estimated using baseline variables identified via double-lasso regression. Estimated propensity scores were used to weight psychiatric hospitalizations in the DREaM OAP-OAP group and compared with observed MMC OAP cohort psychiatric hospitalizations. After the successful calibration of the DREaM OAP-OAP group, similar approaches were taken for the OAP-PP and PP-PP groups to transport DREaM effects to MMC data.

Plain language summary
Paliperidone palmitate (PP) is an injectable medicine used to treat schizophrenia. In the Disease Recovery Evaluation and Modification (DREaM) clinical trial, PP reduced the number of psychiatric hospital visits for patients. In this study, we investigated whether PP could have a similar effect on nontrial patients. We identified a cohort of patients enrolled in Medicaid who shared features with the patients from the DREaM trial. Our analysis showed that PP could reduce psychiatric hospital visits in these patients as well.

Implications for managed care pharmacy
In this calibration of DREaM clinical trial data to a real-world cohort of patients enrolled in Medicaid, treatment strategies that included long-acting injectable PP were predicted to reduce psychiatric hospitalizations over 18 and 30 months in patients with recent-onset schizophrenia vs strategies that included the use of oral antipsychotics. The significant reduction in psychiatric hospitalizations, taken together with the potential reduction of associated health care costs, demonstrates the benefit of including PP in Managed Medicaid formularies.
The Disease Recovery Evaluation and Modification (DREaM) randomized controlled trial (RCT) compared the time to first treatment failure with paliperidone palmitate (PP, a long-acting injectable antipsychotic [LAI]) or oral antipsychotics (OAPs) in patients with recent-onset schizophrenia or schizophreniform disorder. 1 The trial was conducted in 3 parts (Supplementary Figure 1, available in online article). In part I (oral run-in), participants were treated with oral paliperidone or oral risperidone for 2 months. In part II, participants who adequately tolerated OAPs (paliperidone or risperidone in part I) were randomly assigned 2:1 to continue OAP or switch to PP and were followed for 9 months. Patients randomized to OAPs were allowed to switch to other OAPs prospectively. During this time, participants who were adequately treated with PP once-monthly were transitioned to PP once every 3 months. In part III, participants randomly assigned to PP in part II continued PP for another 9 months, whereas participants randomly assigned to OAPs in part II were randomly assigned a second time 1:1 to treatment with OAPs or PP and followed for another 9 months. 1 A previous post hoc analysis using DREaM data (hereafter, we refer to this previous analysis as DREaM post hoc [DPH] analysis) 2 capitalized on these successive randomizations and applied the principal stratification approach to study 3 randomized groups of patients with schizophrenia, each exposed to a different sequential treatment strategy: OAP-OAP, OAP-PP, or PP-PP. The analysis revealed that the OAP-PP and PP-PP strategies generated statistically significant reductions in 18-month cumulative psychiatric hospitalizations (−0.27 [95% CI = −0.50 to −0.04] and −0.28 [95% CI = −0.51 to −0.08]), respectively, compared with the OAP-OAP strategy, primarily because of a faster accumulation of psychiatric hospitalizations in the OAP-OAP strategy compared with the OAP-PP or PP-PP strategies over the 18-month period.
When decision-makers use clinical trial results to inform population-level decisions, they often try to assess the generalizability of these data to real-world populations to anticipate the potential realized value of their choices. In a typical comparative effectiveness study using real-world data, observational data methods are used to compare the outcomes for patients receiving different treatments. However, there are challenges to evidence generation for such purposes. These strategies are often subject to sizeable confounding bias problems. 3 Moreover, when 1 or more treatment strategies are relatively new, real-world data often do not include enough patients receiving these strategies to draw statistical comparisons.
An alternate way to generate real-world evidence for comparative effectiveness is to rely on RCT data and calibrate the RCT results to a control cohort extracted from real-world data. [4][5][6] Such an approach can provide robust real-world evidence early in the adoption timeline of new technologies or therapies. 4,5 Herein, we describe a similar approach to calibrating the comparative effectiveness results from a health care resource utilization analysis of the DREaM study 2 to the target real-world population of patients with recent-onset schizophrenia enrolled in Medicaid Managed Care (MMC).
In the current study, we evaluated whether the results of the DPH analysis were generalizable to the broader real-world population of patients with recent-onset schizophrenia enrolled in MMC and receiving continuous OAP treatment. We also evaluated how the distribution of baseline risk factors in the target population could modify the treatment effect results observed in the DPH study.

DATA AND PATIENT SELECTION
Details of the inclusion/exclusion criteria for the DREaM study have been previously described. 1 Briefly, the study enrolled adult men and women aged 18 to 35 years who had a current diagnosis of schizophrenia or schizophreniform disorder and a first psychotic episode within the previous 24 months. Patients were excluded if they met the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition criteria for a moderate or severe substance use disorder (except for RESULTS: Standardized mean differences in baseline covariates between DREaM treatment arms and MMC groups were substantially reduced after calibration. The 18-month cumulative numbers of psychiatric hospitalizations per patient (SE) were 0.83 (0.14) for the MMC cohort, 0.43 (0.14) for the unweighted OAP-OAP, and 0.80 (0.37) for the calibrated OAP-OAP. The difference between the calibrated OAP-OAP and MMC was not statistically significant (difference, 0.03 [95% CI = −0.67 to 0.81]), indicating successful calibration. The mean difference in 18-month cumulative psychiatric hospitalizations relative to the MMC cohort was -0.77 (95% CI = −1.08 to −0.47) for OAP-PP and −0.83 (95% CI = −1.15 to −0.60) for PP-PP.

CONCLUSIONS:
Our study demonstrates that results from the DREaM OAP-OAP group reflect psychiatric hospitalizations in a real-world population when calibrated using specific baseline characteristics. Transporting the DREaM effects, we find that using OAP-PP and PP-PP treatment strategies for patients with recent-onset schizophrenia in the MMC population could reduce psychiatric hospitalizations compared with the use of OAP. These findings, along with the potential reduction in associated costs, should be considered when assessing the value of PP formulations. nicotine) within 2 months of screening. The present analysis only included patients with a diagnosis of schizophrenia, as that is the US Food and Drug Administration-approved indication for 1-month or 3-month PP extended-release injectable suspension (Invega Sustenna and Invega Trinza, respectively; Janssen Pharmaceuticals, Inc.). 7,8 All study participants provided written informed consent.
We used the 2015 to 2019 MMC dataset provided by IBM MarketScan for the real-world population. This database contains the medical, surgical, and prescription drug experiences of more than 47 million Managed Medicaid enrollees from multiple states. It includes records of inpatient services, inpatient admissions, outpatient services, prescription drug claims, and information about long-term care and other medical care. Data on eligibility (by month), service, provider type, and patient demographics are also included.
The real-world target population for the DPH was identified in the MMC data by applying specific inclusion/ exclusion criteria from the DREaM study ( Figure 1). We identified patients with any diagnosis code of schizophrenia (International Classification of Disease, Ninth Revision [ICD-9]: 295.xx and Tenth Revision [ICD-10]: F20.xx) at any position during 2015 to 2019. We restricted inclusion to those who were enrolled continuously for at least 18 months. Currently, no standard definition for recentonset schizophrenia is published in the literature. We used a working definition of recent-onset schizophrenia that required patients to have continuous enrollment for at least 12 months before the first (index) diagnosis of schizophrenia and to have incurred no utilization with schizophrenia ICD codes at any position between the start of continuous enrollment and the index diagnosis (this period is referred to as the pre-index diagnosis period; Figure 2). Following the DREaM study criteria, patients were not permitted to have any LAI use during the pre-index period. Patients were required to have an OAP prescription (risperidone or paliperidone) within 90 days of the index diagnosis, and this index prescription was continued for at least 60 days, in alignment with the study design of the DREaM trial, in which all patients received OAPs (risperidone or paliperidone) during the oral run-in period (Part I). 1 The end of the 60-day period from index prescription (corresponding to the end of the run-in phase in DREaM) marks the index date for the current study.

EXPOSURE
We compared the 3 treatment strategies (OAP-OAP, OAP-PP, and PP-PP) included in the DPH study. OAP medications were identified in the MMC database using National a This period was defined as the enrollment end date minus the enrollment start date. b The prediagnosis period was defined as the data of index date minus the enrollment start date. Index diagnosis was the first record of a schizophrenia diagnosis within the continuous enrollment period. Continuously enrolled in MMC for >1.5 years a n = 62,025 Not simultaneously enrolled in Medicare n = 45,169 ≥12 months of pre-period before first schizophrenia diagnosis during continuous enrollment b n = 15,461 Age ≥18 to ≤35 years at index date n = 6,039 Used LAI within 12 months of index diagnosis n = 5,356 Used LAI between index diagnosis and enrollment date n = 4,419 Did not initiate OAP within 90 days of index diagnosis date n = 2,149 Did not have ≥60-day continuous supply of index prescription after index prescription date n = 1,124 Did not have ≥90 days of enrollment after index date c n= 1,000

FIGURE 1 Application of DPH Study Inclusion and Exclusion Criteria to the MMC Population
Inclusion Criteria

Exclusion Criteria
hospitalizations, which were the major driver of the treatment failure endpoint in DREaM. 1

VARIABLES AND POTENTIAL CONFOUNDERS
Calibration variables included those collected as part of the DPH study that were also available in the MMC database. These variables included patient age at index date, race, sex, duration of continuous antipsychotic use before the index drug (index Rx in Figure 2), any substance use diagnosis during the 12 months before the index date (pre-index date period), duration of the preperiod (defined as the index date minus the index diagnosis date), type of OAP (risperidone or paliperidone), and utilizations during the preperiod. Utilizations included the number of psychiatric hospitalizations, the cumulative number of hospitalized days (ie, cumulative length of stay), the number of psychiatric emergency department (ED) visits (which were identified via the same diagnosis codes and diagnosis positions used to identify psychiatric hospitalizations), the number of other ED visits, any day treatments (inpatient stays of 0 days), and the total number of ambulatory or outpatient visits.

SAMPLE SIZE
The trial-based sample size for this study was derived from our DPH analysis. 2 We limited the analysis to only US-based Drug Codes. The OAP-OAP exposure represents the control group of the DPH analysis and is the only strategy that we identified in the MMC data to carry out our calibration exercise. Because of the rare (<1%) use of PP in the MMC population during this period, we did not perform a direct comparative evaluation of PP treatment strategies vs OAP in the MMC data. Instead, once we calibrated the outcomes from the OAP-OAP arms of the DPH study and the MMC database, we used the evidence on outcomes from the OAP-PP and PP-PP treatment strategies of the DPH study and transported them to the MMC population. This exercise informed the counterfactual outcomes as to what would have happened if the real-world OAP cohort had instead followed OAP-PP or PP-PP strategies.

OUTCOME
Psychiatric hospitalizations were identified based on ICD-9 codes 295.xx 9 or ICD-10 codes F20.xx, 10 reported in the primary diagnosis field of the hospitalization. These codes are presented in Supplementary Tables 1 Technique   TABLE 1 Real-world calibration and transportability of the Disease Recovery Evaluation and Modification (DREaM) randomized clinical trial in adult Medicaid beneficiaries with recent-onset schizophrenia intervention and the MMC cohort based on the observed baseline covariates (Table 1). Next, we applied a similar calibration exercise to the outcomes. The cumulative number of psychiatric hospitalizations per patient (SE) in the DPH OAP-OAP group was 0.43 (0.14) over 18 months ( Table 2). The corresponding number in the real-world MMC cohort was 0.83 (0.14). After calibration, the cumulative number of psychiatric hospitalizations per patient in the calibrated DPH OAP-OAP group was 0.80 (0.37) over 18 months. The difference in psychiatric hospitalizations per patient in the calibrated DPH OAP-OAP group and the MMC cohort was not statistically significant (−0.03 [95% CI = −0.67 to 0.81]). The difference in the cumulative number of psychiatric hospitalizations per patient in the calibrated DPH OAP-OAP group and the observed MMC cohort at any time during the 18 months was not statistically significant ( Figure 3).

Baseline Demographic and Disease Characteristics of the MMC Cohort and DPH Study Selected by the Double-Lasso
Consequently, we calibrated the DPH OAP-PP and PP-PP group outcomes to the MMC cohort. We found that, although the cumulative incremental effects of OAP-PP and PP-PP groups vs OAP-OAP groups in the DPH analysis were −0.31 (95% CI = −0.63 to −0.04) and −0.38 (95% CI = −0.67 to −0.13), respectively, when calibrated to the MMC cohort, the projected reduction in psychiatric hospitalizations over 18 months from the OAP-PP and PP-PP arms were much higher. We estimated that, compared with the observed MMC cohort receiving OAPs, treating this cohort with OAP-PP or PP-PP would have generated a difference of −0.77 (95% CI = −1.08 to −0.47) or −0.83 (95% CI = −1.15 to −0.60) psychiatric hospitalizations per patient over 18 months, respectively ( Table 2). The results were identical when

Results
Compared with the DPH OAP-OAP group, real-world patients with recent-onset schizophrenia in the MMC data were slightly older (26 vs 22 years), more likely to be female (40% vs 19%) or those who self-identified as non-Hispanic White (40% vs 33%), and less likely to be self-identified Hispanic (3% vs 23%). The MMC patients were also more likely to use OAP for less than 6 months during the pre-index diagnosis period (93% vs 44%) and more likely to have a substance abuse issue in the last 2 years (88% vs 21%) compared with the DPH OAP-OAP treatment group (Table 1). The MMC patients had, on average, more nonpsychiatric ED visits (0.31 vs 0.02) and outpatient visits (0.53 vs 0.35) during the preperiod. The standardized mean differences (SMDs) for these covariates ranged from 0.02 to 1.86 for the comparison of the noncalibrated DPH and MMC cohorts.
The double-lasso method identified the relevant predictors of our propensity score model ( Table 1) TABLE 2 any residual observed confounding, produced almost identical results to the main analysis.

Discussion
In this analysis, we aimed to transport DPH results to a real-world population of patients with recent-onset schizophrenia to estimate the potential real-world impact of the early use of PP, based on the results of the DREaM trial. We found that psychiatric hospitalizations of patients enrolled in the DPH OAP-OAP treatment group could be generalized to the real-world MMC target population through the calibration of selected baseline characteristics. When we transported the DPH treatment effects to the MMC population, we observed even larger reductions in psychiatric hospitalizations among the MMC patients than those estimated in the DPH analysis over 18 months. We next extrapolated the 18-month results to a 30-month follow-up window and found that PP-PP-based regimens can produce statistically significant reductions of up to 1.4 hospitalizations per patient.
Notably, we observed that the absolute risk of psychiatric hospitalizations in the MMC was higher than in the DPH. This may have been due to the impact of enrollment in a clinical trial, in which the increased level of care provided to a patient, including caregiver intervention, may lower the risk of illness exacerbation resulting in acute hospitalization in both treatment arms. Furthermore, the ascertainment of certain comorbidities that may contribute to the risk of acute exacerbation (eg, substance abuse) is likely more reliable in a trial setting than in real-world data. Patients who enroll in clinical trials may also have a higher motivation for self-care and the support of caregivers, which serve as protective factors number for the extrapolated DPH OAP-OAP treatment group after calibration was 1.78 (95% CI = 0.15-3.43), the difference of which was not statistically significant from the observed MMC OAP cohort (difference: 0.33 [95% CI = −1.91 to 1.40]).
After similar extrapolation of the calibrated DPH OAP-PP and PP-PP results from the first 18 months, it was estimated that, compared with the observed MMC cohort receiving OAP, treating them with OAP-PP or PP-PP would have generated a difference of −1.37 (95% CI = −1.86 to −0.94) or −1.45 (95% CI = −1.93 to −1.06) psychiatric hospitalizations per patient over 30 months, respectively (Table 2). Results from the sensitivity analyses, in which covariates were additionally adjusted in the calibrated cohorts to address the temporal trends were alternatively modeled nonparametrically (using quarterly dummies) over 18 months. clinical trial and the real-world sample. However, the similarity of calibrated DPH OAP-OAP outcomes to the observed MMC outcomes serves as a validation that our approach was not duly affected by such unobserved factors. Third, after calibration, some SMDs did not reach the typical benchmark of 0.1 to signal close alignment of those covariate statistics 3 ; this is a manifestation of our small sample sizes, in which perfect alignment between covariates was not possible. However, we do see that calibration improved SMDs for all covariates. Fourth, the small sample size of the DPH study generated uncertainty in the calibration exercise, especially when the temporal trends were extrapolated to 30 months. However, our effect sizes still reach statistical significance because of the large estimates of the effect sizes and, therefore, are likely to signal a genuine reduction in real-world settings. Fifth, linear extrapolation of the projected 18-month reduction in psychiatric hospitalizations was used to predict subsequent observations through 30 months. This extrapolation relies upon the assumption that data trends are maintained throughout the 30-month period. Lastly, there are also limitations in the MMC data source that may have impacted the calibration results. For instance, Hispanic patients were only 3% of the MMC cohort, whereas approximately one-third of patients enrolled in the DREaM trial were Hispanic. However, the calibration was able to significantly reduce the SMD in these patients from 1.11 to 0.06 for the OAP-OAP group.
Despite these limitations, this study provides valuable insights into the potential benefits of LAI use compared with OAPs among patients with recent-onset schizophrenia. As early LAI use continues to become more commonplace, clinicians may find this insight useful while considering the long-term health outcome and health care cost benefits of treatment strategies for patients recently diagnosed with schizophrenia.

Conclusions
Our study demonstrates that the results from the DREaM OAP-OAP group could be calibrated to reflect psychiatric hospitalizations in a real-world population. Transporting the DREaM effects, we find that using OAP-PP and PP-PP treatment strategies for Medicaid beneficiaries with recent-onset schizophrenia could reduce psychiatric hospitalizations. These findings, along with the potential reduction in associated costs, should be considered when assessing the value of treatment options for patients with recent-onset schizophrenia. Such evidence is useful for policymakers to appropriately assess the value of using PP vs OAP in real-world patients with recent-onset schizophrenia. against acute exacerbation. [15][16][17] This raises the question of the generalizability of trial results to real-world data. However, one way to address such concerns is to see if calibration of the control arm of the trial using observed baseline covariates can reproduce observed outcomes in the real-world data. We followed this approach in this work. Successful calibration of the control arm outcomes produces stronger assurances that treatment effects from the trial could be reliably transported to the real-world setting. In our case, the transported effects were bigger in the real-world setting as compared with the DPH study because the type of patients incurring more hospitalizations under the control arm in the DPH study were present in higher proportions in the real-world data.
Our work has implications for future designs of comparative effectiveness studies. Specifically, sampling for a well-defined target population is always helpful for generalizability. Such sampling not only encompasses the clinical inclusion/exclusion criteria but, preferably, targets specific payer systems in which the prevalence of the condition is highest. Also, if transportability analysis such as was used in the current study is planned, collecting a set of baseline covariates in common with those available in real-world data would readily improve the quality of the analyses.
Our prior DPH work has demonstrated that DREaM participants who received PP experienced significant reductions in psychiatric hospitalizations and associated costs compared with OAPs. 2 The current study demonstrates that those trial-based reductions in psychiatric hospitalizations with PP could generalize to a real-world patient cohort and adds to the emerging body of literature supporting LAI use in early schizophrenia. In the DREaM trial and other RCTs, LAIs have been shown to improve outcomes, such as the risk of treatment failure/relapse (including hospitalizations) and the severity of psychotic symptoms, compared with OAPs in patients with recentonset schizophrenia. 1,[18][19][20] Consequently, clinical guidelines, which recommend treatment with LAIs in patients who prefer the injectable formulation and/or are at high risk of nonadherence, increasingly suggest considering LAIs early in the course of illness. [21][22][23][24][25]

LIMITATIONS
The study has some limitations. First, claims data were used to identify recent-onset schizophrenia, and these data may not be inclusive of a patient population with recent-onset schizophrenia. We relied on observed and common patient characteristics between the DPH and MMC data to calibrate the DPH and project outcomes to the real world. Second, this exercise could be incomplete if some unobserved characteristics that affect outcomes are different between the